nvx-rs: add the VMM supervisor with a spawn allowlist, stop, and reattach - #656

Step H3 of #620: one supervisor of a sandbox's VMM, nvx::host::supervisor, beneath the CLI, host::instance, and external orchestrators. It launches OpenVMM through the launch journal of the sandbox store from #628, passes it only the descriptors or handles that the caller allows, waits for its exit by event, stops it through the caller's graceful step with a bounded forced step on request, and reattaches to it after the orchestrator restarts within the same host boot. Callers keep the configuration, the launch planning, the authorization, and the application protocol: the supervisor owns no control session.

What changed

• Spawn allowlist (supervisor::spawn, VmmCommand, Inherited): the caller names the descriptors (Linux) or handles (Windows) that OpenVMM inherits besides its standard streams, which are the capability pipe on standard input and the claimed log on standard output and error. The VMM inherits each one under the number that it has in the caller, so the caller can name it in OpenVMM's arguments, for example as /proc/self/fd/N (Inherited::proc_path), and inherits nothing else of the caller's.
  • On Linux, the descriptors stay close-on-exec in the caller. Between its fork and its exec, the child marks every descriptor close-on-exec, as before, and then clears the flag of the allowed ones only, so a process that another thread spawns meanwhile inherits none of them. A descriptor that is not close-on-exec in the caller, which any process that the caller spawns would inherit, is refused, as are standard streams.
  • On Windows, the allowed handles join the explicit PROC_THREAD_ATTRIBUTE_HANDLE_LIST of platform::spawn_detached, once each, and are inheritable only while the VMM is created: a guard makes each one inheritable and restores its earlier flag right after CreateProcessW, as it does for the standard handles. Spawns that allow handles take that window one at a time, so several may allow the same handle. The job breakaway is unchanged. A process that another thread creates in that window without a handle list, as std::process::Command does, can inherit them too; Inherited documents it.
  • Other hosts refuse any allowed descriptor.
• Launch (supervisor::launch, Starting): runs the store's launch steps under the sandbox's lock: LockedStore::begin_launch, the caller's command for the endpoint that the launch chose, Launch::take_log, the spawn, and Launch::record_process right after it. The caller checks through its own transport that the VMM serves its endpoint; then Starting::commit writes the runtime record and returns a Vmm, or Starting::abandon ends the launch as Launch::abandon does. A step that fails ends the launch, after it ends a VMM whose identity it could not record through the child's handle, and says so when the launch remains for recovery. Dropping a Starting abandons it, and a crash leaves the journal for LockedStore::recover.
• Exit waits (Vmm::wait, Vmm::has_exited): the Process of nvx-rs: add verified process references with event-driven exit waits #625, a pidfd on Linux and the process handle on Windows, with polling only on Linux hosts without pidfds.
• Stop (Vmm::stop): one graceful step, a callback of the caller's that receives the VM and the deadline, then a wait for the VMM's exit event until the deadline. A forced step, only when the caller requests it with a bound, terminates the VMM through its verified handle and waits up to that bound; without it, the stop never signals the VMM. Stopped reports Graceful, Forced, or TimedOut, and the error of a graceful step that failed, which the stop does not wait out. A VMM that exited before the stop skips the graceful step. Once the VMM has exited, the stop clears its runtime record, unless the record names another launch by then. The forced bound is checked like the crate's other timeouts, before anything runs.
• Reattach (supervisor::reattach): checks the caller's runtime identity, ends an interrupted launch with LockedStore::recover, reads the runtime record, holds the VMM with Process::open, and compares identities with RuntimeRecord::check_identity. It reports Live; Mismatched, with the IdentityParts that differ; Exited, with the record that it cleared and the recovery; or Indeterminate, with the reason, keeping the runtime record and the journal of an interrupted launch for a later reattach. Reattach signals no process that it has not verified, though recovery may have terminated a verified VMM of an interrupted launch that has not exited yet. A record of an earlier boot, or one whose process ID now names another process, names a VMM that exited. Reattach doesn't replay Start or MAPS, restore lost streams, or resolve images.
• Recovery and concurrent spawns (store): on Linux, a process holds a copy of each descriptor of its parent from its fork until it executes its program. So a spawn by another thread, such as the launch of another sandbox's VMM, held a launch's claim on its log for that moment, and recovery, or the abandon of a failed launch, reported the launch indeterminate and kept its journal. Recovery now checks a held claim again for up to a second before it concludes that a process of the launch may still run.
• The older helpers keep working until H4 moves the CLI and host::instance, which are unchanged: process::spawn, detach_child, and kill_child go through the supervisor with an empty allowlist, and Launched moved to the supervisor, which process re-exports. The store's documentation now points to the supervisor.

Tests

• nvx-rs/tests/supervisor.rs, which nvx-rs: add verified process references with event-driven exit waits #625 started, also runs the test executable as an orchestrator that launches through the supervisor and crashes at each launch step:
  • before the spawn: reattach reports the launch exited;
  • between the spawn and the identity: a VMM stand-in that serves no endpoint is never signaled, and reattach stays indeterminate until it exits; with the testing feature, the fake OpenVMM, which serves its endpoint, is verified and ended;
  • before the endpoint: the recorded VMM, which isn't the caller's child, is ended through its verified identity.
• A committed VMM outlives its orchestrator; the restarted orchestrator reattaches to it and stops it gracefully, which clears its record. A VMM of another runtime is reported mismatched: a stop without the forced step times out and leaves it running with its record, and a requested forced stop reports forced. Runtime records with a reused process ID or an earlier boot are reported exited, and the process that has the ID isn't signaled.
• The allowlist test launches VMMs that inherit one pipe each, while another thread spawns through the supervisor and through std::process with a pipe that it makes inheritable. On Linux, each VMM holds exactly descriptors 0, 1, and 2, with the log on 1 and 2, and its pipe under the caller's number, and no process of the other thread holds a VMM's pipe. On Windows, each VMM holds its pipe under the caller's value, and none of the handles of the other thread's spawns.
• On Windows: a VMM that breaks away survives its orchestrator's job, one that doesn't ends with it, and a VMM whose identity this user may not query is reported indeterminate and left running.
• Three tests that checked a released lock or claim with a single try, two of the store's and the instance lock test, retry for a bounded time, since a process that a concurrent test spawns holds the lock or claim until it executes its program.
• Unit tests cover the stop decisions against a scripted VMM; the reattach results, with this test process as a held VMM, including an identity that cannot be examined, which the Linux integration tests cannot produce without privileges; the early failures of a launch; the descriptor numbers on Linux; and the inheritance of Windows handles while they are inheritable.

Not in this PR: the delegation of the CLI and host::instance (H4), and the test adapter's host half and the restart acceptance (H5).

Part of #620. Refs #535.

Validation

• Windows 11, x64: python scripts\nvx.py verify and python scripts\check_markdown_links.py. In nvx-rs, with NVX_BUNDLE=skip: cargo fmt --all --check, cargo clippy --locked --all-targets --all-features -- -D warnings, cargo clippy --locked --lib --no-default-features -- -D warnings, cargo test --locked --all-features, cargo test --locked, cargo doc --locked --no-deps --all-features with RUSTDOCFLAGS=-D warnings, and cargo +1.89 check --locked --all-targets --all-features. The same steps for the aci_edge_sandboxes workspace, and python scripts\nvx.py test-agent-bundling --toolchain 1.95.0, as check-aci-edge-sandboxes runs them.
• Linux (Ubuntu on WSL 2): the same steps for both crates, plus cargo check --locked --all-targets --all-features --target aarch64-apple-darwin. The tests ran as an unprivileged user, since the recorded --mount-owner caller scenarios of tests/cli.rs refuse a root-owned share.
• The runtime adapter (tests/runtime-adapter), on both: the steps of check-nvx-rs for its protocol-only and host-only configurations, and, on Linux, for its x86_64-unknown-linux-musl guest-only configuration.
• tests/supervisor.rs passed 10 runs in a row on each OS, on 853ddce. Without the lock of the Windows spawns that allow handles, its new unit test failed 4 runs out of 4.
• Real VMs, WHP (Windows 11), on 8bf8b9a, with NVX_RS naming a release build of it and the newest release that python scripts\nvx.py download --hypervisor whp installs, v0.1.0-dev.dad3e41d7f05: python scripts\nvx.py test-microvm --backend whp, its default scenarios; the same with --scenario sandbox-lifecycle --scenario managed-exec-config, which launch through nvx.py sandbox, and so through the supervisor's spawn; and python scripts\nvx.py test-aci-edge-sandboxes --backend whp, which launches through host::instance. All passed.
• KVM (WSL 2): not run. OpenVMM's --cpu-profile auto selects the built-in profile of this host's CPU, which nested KVM refuses (MSR 0x10a bits 8, 17 are not supported) before any VM boots, and test-microvm takes no host profile.
• The race of the store's tests: on a 24-thread Linux host, the library tests of 853ddce, with --all-features, failed in 20 runs out of 20, against none of 20 at its base, b47a9f8, and none of 8 when they skipped the new host::supervisor tests. A child that a host::supervisor test forked held other tests' locks and logs for about 5 ms before its exec. With the fix, 8bf8b9a passed 20 runs at the default thread count and 20 with 48 test threads, and so did c7b7c94.
• All of the above ran on 8bf8b9a, after the rebase onto dev at 916a61f, past tests: serve the control session in the runtime adapter's guest agent #640, nvx-rs: run the control session on the shared NVXS session's host role #641, and doc: map the OpenVMM upstream roadmap onto the microvm/v2.2 chain #667. The rebase resolved two documentation conflicts with nvx-rs: run the control session on the shared NVXS session's host role #641, in nvx-rs/README.md and nvx-rs/src/host/mod.rs, keeping both sides; the code of the earlier commits is unchanged.
• After the final review, c7b7c94 changes only the instance lock test: on it, the nvx-rs steps above ran again on both OSes, and CI passed on it, in run 39542215794.
• After human review, 3c9deee to 8b0bf90 name the last standard stream descriptor, recheck a held log claim at the store's 25 ms lock retry interval instead of 10 ms, share the unit tests' wait for a released lock, and document process::POLL_INTERVAL. On them, all of the Windows and Linux steps above ran again, and the Linux library tests passed 20 runs at the default thread count and 20 with 48 test threads.
• Left to CI: KVM and MSHV, the WHP matrix on the CI runners, the guest-binary checks, and the bundling check of a synthetic release.